{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c0b97921-3429-44f7-84d3-3229dd48d8e8",
   "metadata": {},
   "source": [
    "# Running custom model training on Vertex AI Pipelines\n",
    "\n",
    "In this lab, you will learn how to run a custom model training job using the Kubeflow Pipelines SDK on Vertex AI Pipelines.\n",
    "\n",
    "## Learning objectives\n",
    "\n",
    "* Use the Kubeflow Pipelines SDK to build scalable ML pipelines.\n",
    "* Create and containerize a custom Scikit-learn model training job that uses Vertex AI managed datasets.\n",
    "* Run a batch prediction job within Vertex AI Pipelines.\n",
    "* Use pre-built components, which are provided through the google_cloud_pipeline_components library, to interact with Vertex AI services.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18a1954a-4054-4f12-bd4b-5ec221cfefc3",
   "metadata": {},
   "source": [
    "## Vertex AI Pipelines setup\n",
    "There are a few additional libraries you'll need to install in order to use Vertex AI Pipelines:\n",
    "\n",
    "* __Kubeflow Pipelines__: This is the SDK you'll be using to build your pipeline. Vertex AI Pipelines supports running pipelines built with both Kubeflow Pipelines or TFX.\n",
    "* __Google Cloud Pipeline Components__: This library provides pre-built components that make it easier to interact with Vertex AI services from your pipeline steps.\n",
    "\n",
    "Each learning objective will correspond to a __#TODO__ in the [student lab notebook](../labs/custom_model_training.ipynb) -- try to complete that notebook first before reviewing this solution notebook.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b3140aa-f652-463e-baed-5946a70dcee6",
   "metadata": {},
   "source": [
    "To install both of the services to be used in this notebook, first set the user flag in the notebook cell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "fdef5d67-f28e-46e4-9c7f-5fabf8b30684",
   "metadata": {},
   "outputs": [],
   "source": [
    "USER_FLAG = \"--user\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1adb84ca-487a-44ff-8969-071553bfee51",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting google-cloud-aiplatform==1.7.0\n",
      "  Downloading google_cloud_aiplatform-1.7.0-py2.py3-none-any.whl (1.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m7.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: packaging>=14.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform==1.7.0) (21.3)\n",
      "Requirement already satisfied: google-cloud-bigquery<3.0.0dev,>=1.15.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform==1.7.0) (2.34.2)\n",
      "Requirement already satisfied: google-api-core[grpc]<3.0.0dev,>=1.26.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform==1.7.0) (2.5.0)\n",
      "Collecting google-cloud-storage<2.0.0dev,>=1.32.0\n",
      "  Downloading google_cloud_storage-1.44.0-py2.py3-none-any.whl (106 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m106.8/106.8 KB\u001b[0m \u001b[31m656.1 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: proto-plus>=1.10.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform==1.7.0) (1.20.3)\n",
      "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.0) (2.27.1)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.52.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.0) (1.54.0)\n",
      "Requirement already satisfied: google-auth<3.0dev,>=1.25.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.0) (2.6.0)\n",
      "Requirement already satisfied: protobuf>=3.12.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.0) (3.19.4)\n",
      "Requirement already satisfied: grpcio<2.0dev,>=1.33.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.0) (1.44.0)\n",
      "Requirement already satisfied: grpcio-status<2.0dev,>=1.33.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.0) (1.44.0)\n",
      "Requirement already satisfied: python-dateutil<3.0dev,>=2.7.2 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform==1.7.0) (2.8.2)\n",
      "Requirement already satisfied: google-cloud-core<3.0.0dev,>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform==1.7.0) (2.2.3)\n",
      "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform==1.7.0) (2.3.2)\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage<2.0.0dev,>=1.32.0->google-cloud-aiplatform==1.7.0) (1.16.0)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging>=14.3->google-cloud-aiplatform==1.7.0) (3.0.7)\n",
      "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.0) (5.0.0)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.0) (4.8)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.0) (0.2.7)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform==1.7.0) (1.1.2)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.0) (2.0.12)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.0) (2021.10.8)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.0) (3.3)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.0) (1.26.8)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform==1.7.0) (1.15.0)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=1.25.0->google-api-core[grpc]<3.0.0dev,>=1.26.0->google-cloud-aiplatform==1.7.0) (0.4.8)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery<3.0.0dev,>=1.15.0->google-cloud-aiplatform==1.7.0) (2.21)\n",
      "Installing collected packages: google-cloud-storage, google-cloud-aiplatform\n",
      "\u001b[33m  WARNING: The script tb-gcp-uploader is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0mSuccessfully installed google-cloud-aiplatform-1.7.0 google-cloud-storage-1.44.0\n",
      "Collecting kfp==1.8.9\n",
      "  Downloading kfp-1.8.9.tar.gz (296 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m296.3/296.3 KB\u001b[0m \u001b[31m11.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting google-cloud-pipeline-components==0.2.0\n",
      "  Downloading google_cloud_pipeline_components-0.2.0-py3-none-any.whl (135 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.0/135.0 KB\u001b[0m \u001b[31m1.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting absl-py<=0.11,>=0.9\n",
      "  Downloading absl_py-0.11.0-py3-none-any.whl (127 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m127.8/127.8 KB\u001b[0m \u001b[31m2.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting PyYAML<6,>=5.3\n",
      "  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m636.6/636.6 KB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: google-cloud-storage<2,>=1.20.0 in ./.local/lib/python3.7/site-packages (from kfp==1.8.9) (1.44.0)\n",
      "Collecting kubernetes<19,>=8.0.0\n",
      "  Downloading kubernetes-18.20.0-py2.py3-none-any.whl (1.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m14.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hCollecting google-api-python-client<2,>=1.7.8\n",
      "  Downloading google_api_python_client-1.12.11-py2.py3-none-any.whl (62 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.1/62.1 KB\u001b[0m \u001b[31m406.2 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting google-auth<2,>=1.6.1\n",
      "  Downloading google_auth-1.35.0-py2.py3-none-any.whl (152 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m152.9/152.9 KB\u001b[0m \u001b[31m1.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting requests-toolbelt<1,>=0.8.0\n",
      "  Downloading requests_toolbelt-0.9.1-py2.py3-none-any.whl (54 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.3/54.3 KB\u001b[0m \u001b[31m520.4 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: cloudpickle<3,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from kfp==1.8.9) (2.0.0)\n",
      "Collecting kfp-server-api<2.0.0,>=1.1.2\n",
      "  Downloading kfp-server-api-1.8.1.tar.gz (54 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m54.2/54.2 KB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting jsonschema<4,>=3.0.1\n",
      "  Downloading jsonschema-3.2.0-py2.py3-none-any.whl (56 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.3/56.3 KB\u001b[0m \u001b[31m7.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: tabulate<1,>=0.8.6 in /opt/conda/lib/python3.7/site-packages (from kfp==1.8.9) (0.8.9)\n",
      "Requirement already satisfied: click<9,>=7.1.2 in /opt/conda/lib/python3.7/site-packages (from kfp==1.8.9) (8.0.4)\n",
      "Collecting Deprecated<2,>=1.2.7\n",
      "  Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB)\n",
      "Collecting strip-hints<1,>=0.1.8\n",
      "  Downloading strip-hints-0.1.10.tar.gz (29 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting docstring-parser<1,>=0.7.3\n",
      "  Downloading docstring_parser-0.13.tar.gz (23 kB)\n",
      "  Installing build dependencies ... \u001b[?25ldone\n",
      "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n",
      "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting kfp-pipeline-spec<0.2.0,>=0.1.13\n",
      "  Downloading kfp_pipeline_spec-0.1.14-py3-none-any.whl (18 kB)\n",
      "Collecting fire<1,>=0.3.1\n",
      "  Downloading fire-0.4.0.tar.gz (87 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m87.7/87.7 KB\u001b[0m \u001b[31m3.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: protobuf<4,>=3.13.0 in /opt/conda/lib/python3.7/site-packages (from kfp==1.8.9) (3.19.4)\n",
      "Collecting uritemplate<4,>=3.0.1\n",
      "  Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)\n",
      "Requirement already satisfied: pydantic<2,>=1.8.2 in /opt/conda/lib/python3.7/site-packages (from kfp==1.8.9) (1.9.0)\n",
      "Collecting typer<1.0,>=0.3.2\n",
      "  Downloading typer-0.4.1-py3-none-any.whl (27 kB)\n",
      "Collecting typing-extensions<4,>=3.7.4\n",
      "  Downloading typing_extensions-3.10.0.2-py3-none-any.whl (26 kB)\n",
      "Requirement already satisfied: google-cloud-aiplatform>=1.4.3 in ./.local/lib/python3.7/site-packages (from google-cloud-pipeline-components==0.2.0) (1.7.0)\n",
      "Collecting google-api-core<2dev,>=1.26.0\n",
      "  Downloading google_api_core-1.31.5-py2.py3-none-any.whl (93 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m93.3/93.3 KB\u001b[0m \u001b[31m9.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting google-cloud-notebooks>=0.4.0\n",
      "  Downloading google_cloud_notebooks-1.2.1-py2.py3-none-any.whl (343 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m344.0/344.0 KB\u001b[0m \u001b[31m32.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from absl-py<=0.11,>=0.9->kfp==1.8.9) (1.16.0)\n",
      "Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from click<9,>=7.1.2->kfp==1.8.9) (4.11.3)\n",
      "Requirement already satisfied: wrapt<2,>=1.10 in /opt/conda/lib/python3.7/site-packages (from Deprecated<2,>=1.2.7->kfp==1.8.9) (1.14.0)\n",
      "Requirement already satisfied: termcolor in /opt/conda/lib/python3.7/site-packages (from fire<1,>=0.3.1->kfp==1.8.9) (1.1.0)\n",
      "Requirement already satisfied: setuptools>=40.3.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.26.0->google-cloud-pipeline-components==0.2.0) (59.8.0)\n",
      "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.26.0->google-cloud-pipeline-components==0.2.0) (2.27.1)\n",
      "Requirement already satisfied: packaging>=14.3 in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.26.0->google-cloud-pipeline-components==0.2.0) (21.3)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.26.0->google-cloud-pipeline-components==0.2.0) (1.54.0)\n",
      "Requirement already satisfied: pytz in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.26.0->google-cloud-pipeline-components==0.2.0) (2021.3)\n",
      "Requirement already satisfied: google-auth-httplib2>=0.0.3 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.7.8->kfp==1.8.9) (0.1.0)\n",
      "Requirement already satisfied: httplib2<1dev,>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from google-api-python-client<2,>=1.7.8->kfp==1.8.9) (0.20.4)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.1->kfp==1.8.9) (4.8)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.1->kfp==1.8.9) (0.2.7)\n",
      "Collecting cachetools<5.0,>=2.0.0\n",
      "  Downloading cachetools-4.2.4-py3-none-any.whl (10 kB)\n",
      "Requirement already satisfied: google-cloud-bigquery<3.0.0dev,>=1.15.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform>=1.4.3->google-cloud-pipeline-components==0.2.0) (2.34.2)\n",
      "Requirement already satisfied: proto-plus>=1.10.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform>=1.4.3->google-cloud-pipeline-components==0.2.0) (1.20.3)\n",
      "Requirement already satisfied: google-api-core[grpc]<3.0.0dev,>=1.26.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform>=1.4.3->google-cloud-pipeline-components==0.2.0) (2.5.0)\n",
      "Requirement already satisfied: google-resumable-media<3.0dev,>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage<2,>=1.20.0->kfp==1.8.9) (2.3.2)\n",
      "Requirement already satisfied: google-cloud-core<3.0dev,>=1.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage<2,>=1.20.0->kfp==1.8.9) (2.2.3)\n",
      "Requirement already satisfied: pyrsistent>=0.14.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp==1.8.9) (0.18.1)\n",
      "Requirement already satisfied: attrs>=17.4.0 in /opt/conda/lib/python3.7/site-packages (from jsonschema<4,>=3.0.1->kfp==1.8.9) (21.4.0)\n",
      "Requirement already satisfied: urllib3>=1.15 in /opt/conda/lib/python3.7/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp==1.8.9) (1.26.8)\n",
      "Requirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp==1.8.9) (2021.10.8)\n",
      "Requirement already satisfied: python-dateutil in /opt/conda/lib/python3.7/site-packages (from kfp-server-api<2.0.0,>=1.1.2->kfp==1.8.9) (2.8.2)\n",
      "Requirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.7/site-packages (from kubernetes<19,>=8.0.0->kfp==1.8.9) (1.3.1)\n",
      "Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from kubernetes<19,>=8.0.0->kfp==1.8.9) (1.3.1)\n",
      "Requirement already satisfied: wheel in /opt/conda/lib/python3.7/site-packages (from strip-hints<1,>=0.1.8->kfp==1.8.9) (0.37.1)\n",
      "Collecting google-api-core[grpc]<3.0.0dev,>=1.26.0\n",
      "  Downloading google_api_core-2.7.1-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.7/114.7 KB\u001b[0m \u001b[31m1.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.7.0-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.7/114.7 KB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.6.1-py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 KB\u001b[0m \u001b[31m12.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.6.0-py2.py3-none-any.whl (114 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.6/114.6 KB\u001b[0m \u001b[31m14.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.4.0-py2.py3-none-any.whl (111 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.7/111.7 KB\u001b[0m \u001b[31m10.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Downloading google_api_core-2.3.2-py2.py3-none-any.whl (109 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m109.8/109.8 KB\u001b[0m \u001b[31m10.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: grpcio<2.0dev,>=1.29.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core<2dev,>=1.26.0->google-cloud-pipeline-components==0.2.0) (1.44.0)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage<2,>=1.20.0->kfp==1.8.9) (1.1.2)\n",
      "Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /opt/conda/lib/python3.7/site-packages (from httplib2<1dev,>=0.15.0->google-api-python-client<2,>=1.7.8->kfp==1.8.9) (3.0.7)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.1->kfp==1.8.9) (0.4.8)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<2dev,>=1.26.0->google-cloud-pipeline-components==0.2.0) (2.0.12)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core<2dev,>=1.26.0->google-cloud-pipeline-components==0.2.0) (3.3)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->click<9,>=7.1.2->kfp==1.8.9) (3.7.0)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib->kubernetes<19,>=8.0.0->kfp==1.8.9) (3.2.0)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage<2,>=1.20.0->kfp==1.8.9) (1.15.0)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=1.3.0->google-cloud-storage<2,>=1.20.0->kfp==1.8.9) (2.21)\n",
      "Building wheels for collected packages: kfp, docstring-parser, fire, kfp-server-api, strip-hints\n",
      "  Building wheel for kfp (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for kfp: filename=kfp-1.8.9-py3-none-any.whl size=409653 sha256=751fb760342f8cceb13306c883c59bc1fec0b9488862b8dc107d3e8bf71e0a3f\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/0e/20/7e/c2c43249eb0538c5aa2542bcc9b02affb0211ed5617fbd4abc\n",
      "  Building wheel for docstring-parser (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for docstring-parser: filename=docstring_parser-0.13-py3-none-any.whl size=31866 sha256=f9f264f772d32d5680cc7a3612ebd733193e05af100610cbc4bd6f7d77cfe798\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/bd/88/3c/d1aa049309f7945178cac9fbe6561a86424f432da57c18ca0f\n",
      "  Building wheel for fire (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for fire: filename=fire-0.4.0-py2.py3-none-any.whl size=115942 sha256=6c35847a17254a9031e00a57dc4545de1abc19f014089b729b9aafc71dcf2f22\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/8a/67/fb/2e8a12fa16661b9d5af1f654bd199366799740a85c64981226\n",
      "  Building wheel for kfp-server-api (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for kfp-server-api: filename=kfp_server_api-1.8.1-py3-none-any.whl size=95549 sha256=7f7aece17e6070b021ee06d32a641dd105ef5ac142363d99468d71ba70b25f44\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/f5/4e/2e/6795bd3ed456a43652e7de100aca275ec179c9a8dfbcc65626\n",
      "  Building wheel for strip-hints (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for strip-hints: filename=strip_hints-0.1.10-py2.py3-none-any.whl size=22302 sha256=5a58f9f2122b6ead8b86af816aaaa239d8e816ca4d56b1b0f8419edcbf5aea89\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/5e/14/c3/6e44e9b2545f2d570b03f5b6d38c00b7534aa8abb376978363\n",
      "Successfully built kfp docstring-parser fire kfp-server-api strip-hints\n",
      "Installing collected packages: typing-extensions, uritemplate, strip-hints, PyYAML, kfp-pipeline-spec, fire, docstring-parser, Deprecated, cachetools, absl-py, requests-toolbelt, kfp-server-api, jsonschema, google-auth, typer, kubernetes, google-api-core, google-api-python-client, google-cloud-notebooks, kfp, google-cloud-pipeline-components\n",
      "\u001b[33m  WARNING: The script strip-hints is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The script jsonschema is installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33m  WARNING: The scripts dsl-compile, dsl-compile-v2 and kfp are installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "tfx 0.26.3 requires absl-py<0.11,>=0.9, but you have absl-py 0.11.0 which is incompatible.\n",
      "tfx 0.26.3 requires attrs<21,>=19.3.0, but you have attrs 21.4.0 which is incompatible.\n",
      "tfx 0.26.3 requires click<8,>=7, but you have click 8.0.4 which is incompatible.\n",
      "tfx 0.26.3 requires docker<5,>=4.1, but you have docker 5.0.3 which is incompatible.\n",
      "tfx 0.26.3 requires kubernetes<12,>=10.0.1, but you have kubernetes 18.20.0 which is incompatible.\n",
      "tfx 0.26.3 requires pyarrow<0.18,>=0.17, but you have pyarrow 7.0.0 which is incompatible.\n",
      "tfx-bsl 0.26.1 requires absl-py<0.11,>=0.9, but you have absl-py 0.11.0 which is incompatible.\n",
      "tfx-bsl 0.26.1 requires pyarrow<0.18,>=0.17, but you have pyarrow 7.0.0 which is incompatible.\n",
      "tensorflow-transform 0.26.0 requires absl-py<0.11,>=0.9, but you have absl-py 0.11.0 which is incompatible.\n",
      "tensorflow-transform 0.26.0 requires pyarrow<0.18,>=0.17, but you have pyarrow 7.0.0 which is incompatible.\n",
      "tensorflow-model-analysis 0.26.1 requires absl-py<0.11,>=0.9, but you have absl-py 0.11.0 which is incompatible.\n",
      "tensorflow-model-analysis 0.26.1 requires pyarrow<0.18,>=0.17, but you have pyarrow 7.0.0 which is incompatible.\n",
      "tensorflow-metadata 0.26.0 requires absl-py<0.11,>=0.9, but you have absl-py 0.11.0 which is incompatible.\n",
      "tensorflow-data-validation 0.26.1 requires absl-py<0.11,>=0.9, but you have absl-py 0.11.0 which is incompatible.\n",
      "tensorflow-data-validation 0.26.1 requires joblib<0.15,>=0.12, but you have joblib 1.0.1 which is incompatible.\n",
      "tensorflow-data-validation 0.26.1 requires pyarrow<0.18,>=0.17, but you have pyarrow 7.0.0 which is incompatible.\n",
      "ml-pipelines-sdk 0.26.3 requires absl-py<0.11,>=0.9, but you have absl-py 0.11.0 which is incompatible.\n",
      "ml-pipelines-sdk 0.26.3 requires docker<5,>=4.1, but you have docker 5.0.3 which is incompatible.\n",
      "ml-metadata 0.26.0 requires absl-py<0.11,>=0.9, but you have absl-py 0.11.0 which is incompatible.\n",
      "ml-metadata 0.26.0 requires attrs<21,>=20.3, but you have attrs 21.4.0 which is incompatible.\n",
      "cloud-tpu-client 0.10 requires google-api-python-client==1.8.0, but you have google-api-python-client 1.12.11 which is incompatible.\n",
      "apache-beam 2.28.0 requires httplib2<0.18.0,>=0.8, but you have httplib2 0.20.4 which is incompatible.\n",
      "apache-beam 2.28.0 requires pyarrow<3.0.0,>=0.15.1, but you have pyarrow 7.0.0 which is incompatible.\n",
      "apache-beam 2.28.0 requires typing-extensions<3.8.0,>=3.7.0, but you have typing-extensions 3.10.0.2 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0mSuccessfully installed Deprecated-1.2.13 PyYAML-5.4.1 absl-py-0.11.0 cachetools-4.2.4 docstring-parser-0.13 fire-0.4.0 google-api-core-1.31.5 google-api-python-client-1.12.11 google-auth-1.35.0 google-cloud-notebooks-1.2.1 google-cloud-pipeline-components-0.2.0 jsonschema-3.2.0 kfp-1.8.9 kfp-pipeline-spec-0.1.14 kfp-server-api-1.8.1 kubernetes-18.20.0 requests-toolbelt-0.9.1 strip-hints-0.1.10 typer-0.4.1 typing-extensions-3.10.0.2 uritemplate-3.0.1\n"
     ]
    }
   ],
   "source": [
    "!pip3 install {USER_FLAG} google-cloud-aiplatform==1.7.0 --upgrade\n",
    "!pip3 install {USER_FLAG} kfp==1.8.9 google-cloud-pipeline-components==0.2.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c6cbece-2a12-45c7-a389-f1b7d0dbbc1e",
   "metadata": {},
   "source": [
    "You may see some warning messages in the install output."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91c6e666-7426-4c7b-adbc-c85a97e0bbc4",
   "metadata": {},
   "source": [
    "After installing these packages you'll need to restart the kernel:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "09739a35-2a55-45da-bd4b-e779aef39f3b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    # Automatically restart kernel after installs\n",
    "    import IPython\n",
    "\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b7b938b-e36f-41af-b307-0005adaf3f3a",
   "metadata": {},
   "source": [
    "Finally, check that you have correctly installed the packages. The KFP SDK version should be >=1.8:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "576366fb-dda2-43f8-a4ac-fa20aa51ac39",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KFP SDK version: 1.8.9\n",
      "google_cloud_pipeline_components version: 0.2.0\n"
     ]
    }
   ],
   "source": [
    "# TODO 1: Check for the KFP SDK version\n",
    "!python3 -c \"import kfp; print('KFP SDK version: {}'.format(kfp.__version__))\"\n",
    "!python3 -c \"import google_cloud_pipeline_components; print('google_cloud_pipeline_components version: {}'.format(google_cloud_pipeline_components.__version__))\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0017776d-e0e8-42ea-a832-bf221eae3eac",
   "metadata": {},
   "source": [
    "### Set your project ID and bucket\n",
    "Throughout this notebook, you'll reference your Cloud project ID and the bucket you created earlier. Next you'll create variables for each of those.\n",
    "\n",
    "If you don't know your project ID you may be able to get it by running the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8f45cde1-30ac-4a6e-bb74-ea27d1997f89",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project ID:  qwiklabs-gcp-03-829279d2a7be\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "PROJECT_ID = \"\"\n",
    "\n",
    "# Get your Google Cloud project ID from gcloud\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
    "    PROJECT_ID = shell_output[0]\n",
    "    print(\"Project ID: \", PROJECT_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f82f158c-b6f9-41bb-8a72-c069f84575a8",
   "metadata": {},
   "source": [
    "Otherwise, set it here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "49999f56-5e1c-412f-933b-23dafffb18f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "if PROJECT_ID == \"\" or PROJECT_ID is None:\n",
    "    PROJECT_ID = \"your-project-id\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab37c893-8827-4f31-aa27-23da273c81ab",
   "metadata": {},
   "source": [
    "Then create a variable to store your bucket name. If you created it in this lab, the following will work. Otherwise, you'll need to set this manually:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c0565c74-4b29-400a-b1e4-dfb693e2ce71",
   "metadata": {},
   "outputs": [],
   "source": [
    "BUCKET_NAME=\"gs://\" + PROJECT_ID + \"-bucket\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1aceea9f-6093-4b12-a82e-039e460a3818",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gs://qwiklabs-gcp-03-829279d2a7be-bucket\n"
     ]
    }
   ],
   "source": [
    "!echo {BUCKET_NAME}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e4d3ab8-5764-4b3c-a0f6-b7e93d1668c3",
   "metadata": {},
   "source": [
    "### Import libraries\n",
    "Add the following to import the libraries you'll be using throughout this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1f060731-9617-4ba1-b740-2f5e76dd5143",
   "metadata": {},
   "outputs": [],
   "source": [
    "from kfp.v2 import compiler, dsl\n",
    "from kfp.v2.dsl import pipeline\n",
    "\n",
    "from google.cloud import aiplatform\n",
    "from google_cloud_pipeline_components import aiplatform as gcc_aip"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73796d00-9388-4f16-8eed-52c42eb9f830",
   "metadata": {},
   "source": [
    "### Define constants\n",
    "The last thing you need to do before building your pipeline is define some constant variables. `PIPELINE_ROOT` is the Cloud Storage path where the artifacts created by your pipeline will be written. You're using `us-central1` as the region here, but if you used a different region when you created your bucket, update the `REGION` variable in the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ee217079-0778-4e50-96dd-e6e8c3ffc156",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: PATH=/usr/local/cuda/bin:/opt/conda/bin:/opt/conda/condabin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/home/jupyter/.local/bin\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'gs://qwiklabs-gcp-03-829279d2a7be-bucket/pipeline_root/'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "PATH=%env PATH\n",
    "%env PATH={PATH}:/home/jupyter/.local/bin\n",
    "REGION=\"us-central1\"\n",
    "\n",
    "PIPELINE_ROOT = f\"{BUCKET_NAME}/pipeline_root/\"\n",
    "PIPELINE_ROOT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "137ed711-f4a1-4535-b4c9-3ca2ae965d8c",
   "metadata": {},
   "source": [
    "After running the code above, you should see the root directory for your pipeline printed. This is the Cloud Storage location where the artifacts from your pipeline will be written. It will be in the format of `gs://YOUR-BUCKET-NAME/pipeline_root/`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddda756e-865f-49b5-866d-92de9aa6db61",
   "metadata": {},
   "source": [
    "## Configuring a custom model training job\n",
    "Before you set up your pipeline, you need to write the code for your custom model training job. To train the model, you'll use the UCI Machine Learning [Dry beans dataset](https://archive.ics.uci.edu/ml/datasets/Dry+Bean+Dataset), from: KOKLU, M. and OZKAN, I.A., (2020), \"Multiclass Classification of Dry Beans Using Computer Vision and Machine Learning Techniques.\"In Computers and Electronics in Agriculture, 174, 105507. [DOI](https://www.sciencedirect.com/science/article/abs/pii/S0168169919311573?via%3Dihub).\n",
    "\n",
    "Your first pipeline step will create a managed dataset in Vertex AI using a BigQuery table that contains a version of this beans data. The dataset will be passed as input to your training job. In your training code, you'll have access to environment variable to access this managed dataset.\n",
    "\n",
    "Here's how you'll set up your custom training job:\n",
    "\n",
    "* Write a Scikit-learn `DecisionTreeClassifier` model to classify bean types in your data.\n",
    "* Package the training code in a Docker container and push it to Container Registry\n",
    "\n",
    "From there, you'll be able to start a Vertex AI Training job directly from your pipeline. Let's get started!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e6ebb84-3ac1-4e4d-8532-d5ec9051f386",
   "metadata": {},
   "source": [
    "### Define your training code in a Docker container\n",
    "Run the following to set up a directory where you'll add your containerized code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "d00754a8-6ecb-46fe-92ea-437f59140d86",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir traincontainer\n",
    "!touch traincontainer/Dockerfile\n",
    "!mkdir traincontainer/trainer\n",
    "!touch traincontainer/trainer/train.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84f023fb-32ed-486a-9fcd-079c4a081332",
   "metadata": {},
   "source": [
    "After running those commands, you should see a directory called traincontainer/ created on the left (you may need to click the refresh icon to see it). You'll see the following in your traincontainer/ directory:\n",
    "\n",
    "```\n",
    "+ Dockerfile\n",
    "+ trainer/\n",
    "    + train.py\n",
    "```\n",
    "Your first step in containerizing your code is to create a Dockerfile. In your Dockerfile you'll include all the commands needed to run your image. It'll install all the libraries you're using and set up the entry point for your training code. Run the following to create a Dockerfile file locally in your notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "e731f273-c74b-4939-ae43-a2c3e685230c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting traincontainer/Dockerfile\n"
     ]
    }
   ],
   "source": [
    "%%writefile traincontainer/Dockerfile\n",
    "FROM gcr.io/deeplearning-platform-release/sklearn-cpu.0-23\n",
    "WORKDIR /\n",
    "\n",
    "# Copies the trainer code to the docker image.\n",
    "COPY trainer /trainer\n",
    "\n",
    "RUN pip install sklearn google-cloud-bigquery joblib pandas google-cloud-storage\n",
    "\n",
    "# Sets up the entry point to invoke the trainer.\n",
    "ENTRYPOINT [\"python\", \"-m\", \"trainer.train\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "163878fd-da68-4ba0-8c92-61693c28811c",
   "metadata": {},
   "source": [
    "Run the following to create `train.py` file. This retrieves the data from your managed dataset, puts it into a Pandas DataFrame, trains a Scikit-learn model, and uploads the trained model to Cloud Storage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "85393be7-16ba-4c41-879a-974f4da96e68",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting traincontainer/trainer/train.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile traincontainer/trainer/train.py\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.model_selection import train_test_split\n",
    "from google.cloud import bigquery\n",
    "from google.cloud import storage\n",
    "from joblib import dump\n",
    "\n",
    "import os\n",
    "import pandas as pd\n",
    "\n",
    "bqclient = bigquery.Client()\n",
    "storage_client = storage.Client()\n",
    "\n",
    "def download_table(bq_table_uri: str):\n",
    "    prefix = \"bq://\"\n",
    "    if bq_table_uri.startswith(prefix):\n",
    "        bq_table_uri = bq_table_uri[len(prefix):]\n",
    "\n",
    "    table = bigquery.TableReference.from_string(bq_table_uri)\n",
    "    rows = bqclient.list_rows(\n",
    "        table,\n",
    "    )\n",
    "    return rows.to_dataframe(create_bqstorage_client=False)\n",
    "\n",
    "# These environment variables are from Vertex AI managed datasets\n",
    "training_data_uri = os.environ[\"AIP_TRAINING_DATA_URI\"]\n",
    "test_data_uri = os.environ[\"AIP_TEST_DATA_URI\"]\n",
    "\n",
    "# Download data into Pandas DataFrames, split into train / test\n",
    "df = download_table(training_data_uri)\n",
    "test_df = download_table(test_data_uri)\n",
    "labels = df.pop(\"Class\").tolist()\n",
    "data = df.values.tolist()\n",
    "test_labels = test_df.pop(\"Class\").tolist()\n",
    "test_data = test_df.values.tolist()\n",
    "\n",
    "# TODO 2: Define and train the Scikit model\n",
    "skmodel = DecisionTreeClassifier()\n",
    "skmodel.fit(data, labels)\n",
    "score = skmodel.score(test_data, test_labels)\n",
    "print('accuracy is:',score)\n",
    "\n",
    "# Save the model to a local file\n",
    "dump(skmodel, \"model.joblib\")\n",
    "\n",
    "# Upload the saved model file to GCS\n",
    "bucket = storage_client.get_bucket(\"YOUR_GCS_BUCKET\")\n",
    "model_directory = os.environ[\"AIP_MODEL_DIR\"]\n",
    "storage_path = os.path.join(model_directory, \"model.joblib\")\n",
    "blob = storage.blob.Blob.from_string(storage_path, client=storage_client)\n",
    "blob.upload_from_filename(\"model.joblib\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d0428e9f-9c5e-4721-8b4c-213820f0b87d",
   "metadata": {},
   "source": [
    "Run the following to replace YOUR_GCS_BUCKET from the script above with the name of your Cloud Storage bucket:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "b74d2b3e-83ae-4fc0-b807-6338893d6f4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "BUCKET = BUCKET_NAME[5:] # Trim the 'gs://' before adding to train script\n",
    "!sed -i -r 's@YOUR_GCS_BUCKET@'\"$BUCKET\"'@' traincontainer/trainer/train.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "380c9e27-ca41-4ef0-818c-edb1fbb70e72",
   "metadata": {},
   "source": [
    "You can also do this manually if you'd prefer. If you do, make sure not to include the gs:// in your bucket name when you update the script.\n",
    "\n",
    "Now your training code is in a Docker container and you're ready to run training in the Cloud."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e06a42c8-91f8-4146-98fb-4e0acddadf38",
   "metadata": {},
   "source": [
    "### Push container to Container Registry\n",
    "\n",
    "With your training code complete, you're ready to push this to Google Container Registry. Later when you configure the training component of your pipeline, you'll point Vertex AI Pipelines at this container.\n",
    "\n",
    "Replace `YOUR_PROJECT_ID` with your PROJECT_ID in the IMAGE_URI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "33898ee1-8edf-44f7-be38-922d5225b9ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "!PROJECT_ID=$(gcloud config get-value project)\n",
    "!IMAGE_URI=\"gcr.io/YOUR_PROJECT_ID/scikit:v1\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a80525c-321a-433d-ad85-ba1dd6fc67d0",
   "metadata": {},
   "source": [
    "Again, replace `YOUR_PROJECT_ID` with your PROJECT_ID and build your container by running the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "817aaecf-ce37-4116-9e87-f4bb8465a513",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sending build context to Docker daemon  9.728kB\n",
      "Step 1/5 : FROM gcr.io/deeplearning-platform-release/sklearn-cpu.0-23\n",
      "latest: Pulling from deeplearning-platform-release/sklearn-cpu.0-23\n",
      "\n",
      "\u001b[1B88808835: Pulling fs layer \n",
      "\u001b[1Bcf64a9ea: Pulling fs layer \n",
      "\u001b[1Bc2aa40d3: Pulling fs layer \n",
      "\u001b[1B2e264020: Pulling fs layer \n",
      "\u001b[1Bb700ef54: Pulling fs layer \n",
      "\u001b[1B65f0ab42: Pulling fs layer \n",
      "\u001b[1Bad4dbd7d: Pulling fs layer \n",
      "\u001b[1B609522d8: Pulling fs layer \n",
      "\u001b[1B2168e631: Pulling fs layer \n",
      "\u001b[1Bbb01bc78: Pulling fs layer \n",
      "\u001b[1Bda654f5c: Pulling fs layer \n",
      "\u001b[1B40a4f176: Pulling fs layer \n",
      "\u001b[1Beec75c0c: Pulling fs layer \n",
      "\u001b[1B728eb7d7: Pulling fs layer \n",
      "\u001b[1B30f752a4: Pulling fs layer \n",
      "\u001b[1B5f7af5b1: Pulling fs layer \n",
      "\u001b[1Bb2dca45b: Pulling fs layer \n",
      "\u001b[1B864a46df: Pulling fs layer \n",
      "\u001b[1B31a7db85: Pulling fs layer \n",
      "\u001b[1Badbf67c9: Pull complete 403kB/1.403kBB20A\u001b[2K\u001b[17A\u001b[2K\u001b[20A\u001b[2K\u001b[17A\u001b[2K\u001b[20A\u001b[2K\u001b[20A\u001b[2K\u001b[18A\u001b[2K\u001b[16A\u001b[2K\u001b[18A\u001b[2K\u001b[17A\u001b[2K\u001b[18A\u001b[2K\u001b[17A\u001b[2K\u001b[20A\u001b[2K\u001b[17A\u001b[2K\u001b[14A\u001b[2K\u001b[17A\u001b[2K\u001b[14A\u001b[2K\u001b[17A\u001b[2K\u001b[14A\u001b[2K\u001b[17A\u001b[2K\u001b[14A\u001b[2K\u001b[20A\u001b[2K\u001b[14A\u001b[2K\u001b[17A\u001b[2K\u001b[14A\u001b[2K\u001b[17A\u001b[2K\u001b[14A\u001b[2K\u001b[17A\u001b[2K\u001b[14A\u001b[2K\u001b[17A\u001b[2K\u001b[14A\u001b[2K\u001b[17A\u001b[2K\u001b[14A\u001b[2K\u001b[17A\u001b[2K\u001b[20A\u001b[2K\u001b[17A\u001b[2K\u001b[14A\u001b[2K\u001b[17A\u001b[2K\u001b[14A\u001b[2K\u001b[20A\u001b[2K\u001b[14A\u001b[2K\u001b[20A\u001b[2K\u001b[14A\u001b[2K\u001b[20A\u001b[2K\u001b[14A\u001b[2K\u001b[20A\u001b[2K\u001b[17A\u001b[2K\u001b[20A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[14A\u001b[2K\u001b[18A\u001b[2K\u001b[14A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[8A\u001b[2K\u001b[19A\u001b[2K\u001b[6A\u001b[2K\u001b[18A\u001b[2K\u001b[3A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[4A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[18A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[16A\u001b[2K\u001b[16A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[14A\u001b[2K\u001b[13A\u001b[2K\u001b[12A\u001b[2K\u001b[11A\u001b[2K\u001b[10A\u001b[2K\u001b[9A\u001b[2K\u001b[8A\u001b[2K\u001b[7A\u001b[2K\u001b[6A\u001b[2K\u001b[5A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[4A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2KDigest: sha256:1c3c7d461744e5ed157f29b69d4b31c991472519b4a88a22cd339498b6522754\n",
      "Status: Downloaded newer image for gcr.io/deeplearning-platform-release/sklearn-cpu.0-23:latest\n",
      " ---> 73bbcb0663a7\n",
      "Step 2/5 : WORKDIR /\n",
      " ---> Running in bfa337ea0367\n",
      "Removing intermediate container bfa337ea0367\n",
      " ---> 54ba54c2d7bd\n",
      "Step 3/5 : COPY trainer /trainer\n",
      " ---> 3a9aa86cca24\n",
      "Step 4/5 : RUN pip install sklearn google-cloud-bigquery joblib pandas google-cloud-storage\n",
      " ---> Running in ab32595cc58a\n",
      "Collecting sklearn\n",
      "  Downloading sklearn-0.0.tar.gz (1.1 kB)\n",
      "  Preparing metadata (setup.py): started\n",
      "  Preparing metadata (setup.py): finished with status 'done'\n",
      "Requirement already satisfied: google-cloud-bigquery in /opt/conda/lib/python3.7/site-packages (2.34.2)\n",
      "Requirement already satisfied: joblib in /opt/conda/lib/python3.7/site-packages (1.1.0)\n",
      "Requirement already satisfied: pandas in /opt/conda/lib/python3.7/site-packages (1.3.5)\n",
      "Requirement already satisfied: google-cloud-storage in /opt/conda/lib/python3.7/site-packages (2.2.1)\n",
      "Requirement already satisfied: scikit-learn in /opt/conda/lib/python3.7/site-packages (from sklearn) (0.23.2)\n",
      "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery) (2.27.1)\n",
      "Requirement already satisfied: google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery) (2.5.0)\n",
      "Requirement already satisfied: grpcio<2.0dev,>=1.38.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery) (1.44.0)\n",
      "Requirement already satisfied: protobuf>=3.12.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery) (3.19.4)\n",
      "Requirement already satisfied: google-resumable-media<3.0dev,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery) (2.3.2)\n",
      "Requirement already satisfied: google-cloud-core<3.0.0dev,>=1.4.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery) (2.2.3)\n",
      "Requirement already satisfied: proto-plus>=1.15.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery) (1.20.3)\n",
      "Requirement already satisfied: packaging>=14.3 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery) (21.3)\n",
      "Requirement already satisfied: python-dateutil<3.0dev,>=2.7.2 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.7/site-packages (from pandas) (2021.3)\n",
      "Requirement already satisfied: numpy>=1.17.3 in /opt/conda/lib/python3.7/site-packages (from pandas) (1.19.5)\n",
      "Requirement already satisfied: google-auth<3.0dev,>=1.25.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-storage) (2.6.0)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.52.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-bigquery) (1.54.0)\n",
      "Requirement already satisfied: grpcio-status<2.0dev,>=1.33.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5->google-cloud-bigquery) (1.44.0)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-cloud-storage) (4.8)\n",
      "Requirement already satisfied: six>=1.9.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-cloud-storage) (1.16.0)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-cloud-storage) (0.2.7)\n",
      "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<3.0dev,>=1.25.0->google-cloud-storage) (5.0.0)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery) (1.1.2)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging>=14.3->google-cloud-bigquery) (3.0.7)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-bigquery) (1.26.8)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-bigquery) (2021.10.8)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-bigquery) (2.0.12)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-cloud-bigquery) (3.3)\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->sklearn) (3.1.0)\n",
      "Requirement already satisfied: scipy>=0.19.1 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->sklearn) (1.7.3)\n",
      "Requirement already satisfied: cffi>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery) (1.15.0)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0dev,>=1.25.0->google-cloud-storage) (0.4.8)\n",
      "Requirement already satisfied: pycparser in /opt/conda/lib/python3.7/site-packages (from cffi>=1.0.0->google-crc32c<2.0dev,>=1.0->google-resumable-media<3.0dev,>=0.6.0->google-cloud-bigquery) (2.21)\n",
      "Building wheels for collected packages: sklearn\n",
      "  Building wheel for sklearn (setup.py): started\n",
      "  Building wheel for sklearn (setup.py): finished with status 'done'\n",
      "  Created wheel for sklearn: filename=sklearn-0.0-py2.py3-none-any.whl size=1310 sha256=c597db554f24a2aaefd4f59a017e0f13895ab98682c8a223e680d490acba43bb\n",
      "  Stored in directory: /root/.cache/pip/wheels/46/ef/c3/157e41f5ee1372d1be90b09f74f82b10e391eaacca8f22d33e\n",
      "Successfully built sklearn\n",
      "Installing collected packages: sklearn\n",
      "Successfully installed sklearn-0.0\n",
      "\u001b[91mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\n",
      "\u001b[0mRemoving intermediate container ab32595cc58a\n",
      " ---> b1a44343c415\n",
      "Step 5/5 : ENTRYPOINT [\"python\", \"-m\", \"trainer.train\"]\n",
      " ---> Running in b57139dfce79\n",
      "Removing intermediate container b57139dfce79\n",
      " ---> 019c848a6060\n",
      "Successfully built 019c848a6060\n",
      "Successfully tagged gcr.io/qwiklabs-gcp-03-829279d2a7be/scikit:v1\n"
     ]
    }
   ],
   "source": [
    "!docker build ./traincontainer -t gcr.io/YOUR_PROJECT_ID/scikit:v1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d078535-72b4-405a-b4ac-e64ca2dffea9",
   "metadata": {},
   "source": [
    "Finally, push the container to Container Registry:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "82e8c491-f943-48c5-8807-98a6eda9e3d0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The push refers to repository [gcr.io/qwiklabs-gcp-03-829279d2a7be/scikit]\n",
      "\n",
      "\u001b[1B680fc625: Preparing \n",
      "\u001b[1B1b61b640: Preparing \n",
      "\u001b[1B4f8407f3: Preparing \n",
      "\u001b[1B368e27aa: Preparing \n",
      "\u001b[1Bdd2b9e38: Preparing \n",
      "\u001b[1Be29d8d24: Preparing \n",
      "\u001b[1B95a574c8: Preparing \n",
      "\u001b[1B10151b48: Preparing \n",
      "\u001b[1Bc089358e: Preparing \n",
      "\u001b[1B9b36546a: Preparing \n",
      "\u001b[1B82ce8d0b: Preparing \n",
      "\u001b[1B467ac3a5: Preparing \n",
      "\u001b[1B91c31559: Preparing \n",
      "\u001b[1Bae11254c: Preparing \n",
      "\u001b[1B2bcbe281: Preparing \n",
      "\u001b[1B4c112e39: Preparing \n",
      "\u001b[1B048fd290: Preparing \n",
      "\u001b[1Bbf18a086: Preparing \n",
      "\u001b[1B7a45d8d8: Preparing \n",
      "\u001b[1B6651fb01: Preparing \n",
      "\u001b[1Bd5cafaa0: Preparing \n",
      "\u001b[2Bd5cafaa0: Mounted from deeplearning-platform-release/sklearn-cpu.0-23 \u001b[18A\u001b[2K\u001b[19A\u001b[2K\u001b[17A\u001b[2K\u001b[21A\u001b[2K\u001b[16A\u001b[2K\u001b[15A\u001b[2K\u001b[13A\u001b[2K\u001b[12A\u001b[2K\u001b[10A\u001b[2K\u001b[11A\u001b[2K\u001b[9A\u001b[2K\u001b[5A\u001b[2K\u001b[8A\u001b[2K\u001b[6A\u001b[2K\u001b[3A\u001b[2K\u001b[4A\u001b[2K\u001b[1A\u001b[2K\u001b[2A\u001b[2Kv1: digest: sha256:4bc18a9be14b00f020df23d14b668172bf91f9f831494509a4d77e550ffce3f4 size: 4916\n"
     ]
    }
   ],
   "source": [
    "!docker push gcr.io/$PROJECT_ID/scikit:v1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31ca7383-8662-44c7-b166-0499a9711d30",
   "metadata": {},
   "source": [
    "Navigate to the [Container Registry section](https://console.cloud.google.com/gcr/) of your Cloud console to verify your container is there."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b221379f-f8c6-45e2-b697-6209a0f9a397",
   "metadata": {},
   "source": [
    "## Configuring a batch prediction job\n",
    "The last step of your pipeline will run a batch prediction job. For this to work, you need to provide a CSV file in Cloud Storage that contains the examples you want to get predictions on. You'll create this CSV file in your notebook and copy it to Cloud Storage using the `gcloud storage` command line tool.\n",
    "\n",
    "### Copying batch prediction examples to Cloud Storage\n",
    "The following file contains 3 examples from each class in your beans dataset. The example below doesn't include the `Class` column since that is what your model will be predicting. Run the following to create this CSV file locally in your notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "id": "174aa58e-3eee-4352-9c91-591bfa6a1afb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing batch_examples.csv\n"
     ]
    }
   ],
   "source": [
    "%%writefile batch_examples.csv\n",
    "Area,Perimeter,MajorAxisLength,MinorAxisLength,AspectRation,Eccentricity,ConvexArea,EquivDiameter,Extent,Solidity,roundness,Compactness,ShapeFactor1,ShapeFactor2,ShapeFactor3,ShapeFactor4\n",
    "23288,558.113,207.567738,143.085693,1.450653336,0.7244336162,23545,172.1952453,0.8045881703,0.9890847314,0.9395021523,0.8295857874,0.008913077034,0.002604069884,0.6882125787,0.9983578734\n",
    "23689,575.638,205.9678003,146.7475015,1.403552348,0.7016945718,24018,173.6714472,0.7652721693,0.9863019402,0.8983750474,0.8431970773,0.00869465998,0.002711119968,0.7109813112,0.9978994889\n",
    "23727,559.503,189.7993849,159.3717704,1.190922235,0.5430731512,24021,173.8106863,0.8037601626,0.9877607094,0.952462433,0.9157600082,0.007999299741,0.003470231343,0.8386163926,0.9987269085\n",
    "31158,641.105,212.0669751,187.1929601,1.132879009,0.4699241567,31474,199.1773023,0.7813134733,0.989959967,0.9526231013,0.9392188582,0.0068061806,0.003267009878,0.8821320637,0.9993488983\n",
    "32514,649.012,221.4454899,187.1344232,1.183349841,0.5346736437,32843,203.4652564,0.7849831,0.9899826447,0.9700068737,0.9188051492,0.00681077351,0.002994124691,0.8442029022,0.9989873701\n",
    "33078,659.456,235.5600775,178.9312328,1.316483846,0.6503915309,33333,205.2223615,0.7877214708,0.9923499235,0.9558229607,0.8712102818,0.007121351881,0.002530662194,0.7590073551,0.9992209221\n",
    "33680,683.09,256.203255,167.9334938,1.525623324,0.7552213942,34019,207.081404,0.80680321,0.9900349805,0.9070392732,0.8082699962,0.007606985006,0.002002710402,0.6533003868,0.9966903078\n",
    "33954,716.75,277.3684803,156.3563259,1.773951126,0.825970469,34420,207.9220419,0.7994819873,0.9864613597,0.8305492781,0.7496238998,0.008168948587,0.001591181142,0.5619359911,0.996846984\n",
    "36322,719.437,272.0582306,170.8914975,1.591993952,0.7780978465,36717,215.0502424,0.7718560075,0.9892420405,0.8818487005,0.7904566678,0.007490177594,0.001803782407,0.6248217437,0.9947124371\n",
    "36675,742.917,285.8908964,166.8819538,1.713132487,0.8119506999,37613,216.0927123,0.7788277766,0.9750618137,0.8350248381,0.7558572692,0.0077952528,0.001569528272,0.5713202115,0.9787472145\n",
    "37454,772.679,297.6274753,162.1493177,1.835514817,0.8385619338,38113,218.3756257,0.8016695205,0.9827093118,0.7883332637,0.7337213257,0.007946480356,0.001420623993,0.5383469838,0.9881438654\n",
    "37789,766.378,313.5680678,154.3409867,2.031657789,0.8704771226,38251,219.3500608,0.7805870567,0.9879218844,0.8085170916,0.6995293312,0.008297866252,0.001225659709,0.4893412853,0.9941740339\n",
    "47883,873.536,327.9986493,186.5201272,1.758516115,0.822571799,48753,246.9140116,0.7584464543,0.9821549443,0.7885506623,0.7527897207,0.006850002074,0.00135695419,0.5666923636,0.9965376533\n",
    "49777,861.277,300.7570338,211.6168613,1.42123379,0.7105823885,50590,251.7499649,0.8019106536,0.9839296304,0.843243269,0.8370542883,0.00604208839,0.001829706116,0.7006598815,0.9958014989\n",
    "49882,891.505,357.1890036,179.8346914,1.986207449,0.8640114945,51042,252.0153467,0.7260210171,0.9772736178,0.7886896753,0.7055518063,0.007160679276,0.001094585314,0.4978033513,0.9887407248\n",
    "53249,919.923,325.3866286,208.9174205,1.557489212,0.7666552108,54195,260.3818974,0.6966846347,0.9825445152,0.7907120655,0.8002231025,0.00611066177,0.001545654241,0.6403570138,0.9973491406\n",
    "61129,964.969,369.3481688,210.9473449,1.750902193,0.8208567513,61796,278.9836198,0.7501135067,0.9892064211,0.8249553283,0.7553404711,0.006042110436,0.001213219664,0.5705392272,0.9989583843\n",
    "61918,960.372,353.1381442,224.0962377,1.575832543,0.7728529173,62627,280.7782864,0.7539207091,0.9886790043,0.8436218213,0.7950947556,0.005703319619,0.00140599258,0.6321756704,0.9962029945\n",
    "141953,1402.05,524.2311633,346.3974998,1.513380332,0.7505863011,143704,425.1354762,0.7147107987,0.9878152313,0.9074598849,0.8109694843,0.003692991084,0.0009853172185,0.6576715044,0.9953071199\n",
    "145285,1440.991,524.9567463,353.0769977,1.486805285,0.7400216694,146709,430.0960442,0.7860466375,0.9902937107,0.8792413513,0.8192980608,0.003613289371,0.001004269363,0.6712493125,0.9980170255\n",
    "146153,1476.383,526.1933264,356.528288,1.475881001,0.7354662103,149267,431.3789276,0.7319360978,0.9791380546,0.8425962592,0.8198107159,0.003600290972,0.001003163512,0.6720896099,0.991924286"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a12325be-c663-4ed8-9e38-df6d7d02e0ce",
   "metadata": {},
   "source": [
    "Then, copy the file to your Cloud Storage bucket:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "9a0ae6a8-e1d6-484e-b5d5-d71907137df5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Copying file://batch_examples.csv [Content-Type=text/csv]...\n",
      "/ [1 files][  4.0 KiB/  4.0 KiB]                                                \n",
      "Operation completed over 1 objects/4.0 KiB.                                      \n"
     ]
    }
   ],
   "source": [
    "!gcloud storage cp batch_examples.csv $BUCKET_NAME"   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fbe3a93-cd8e-4a38-99b6-787c002e62eb",
   "metadata": {},
   "source": [
    "You'll reference this file in the next step when you define your pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4a95a90-fe72-411b-b686-98d7db37e129",
   "metadata": {},
   "source": [
    "### Building a pipeline with pre-built components\n",
    "Now that your training code is in the cloud, you're ready to call it from your pipeline. The pipeline you'll define will use three pre-built components from the `google_cloud_pipeline_components` library you installed earlier. These predefined components simplify the code you need to write to set up your pipeline, and will allow us to use Vertex AI services like model training and batch prediction.\n",
    "\n",
    "If you can't find a pre-built component for the task you want to accomplish, you can define your own Python-based custom components. To see an example, check out [this codelab](https://codelabs.developers.google.com/vertex-pipelines-intro#5).\n",
    "\n",
    "Here's what your three-step pipeline will do:\n",
    "\n",
    "* Create a managed dataset in Vertex AI.\n",
    "* Run a training job on Vertex AI using the custom container you set up.\n",
    "* Run a batch prediction job on your trained Scikit-learn classification model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e61b1f86-7314-4dab-9b66-9fafcf9c03c8",
   "metadata": {},
   "source": [
    "### Define your pipeline\n",
    "Because you're using pre-built components, you can set up your entire pipeline in the pipeline definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "801a633f-5f8e-4ced-8bd9-803787c029f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "@pipeline(name=\"automl-beans-custom\",\n",
    "                  pipeline_root=PIPELINE_ROOT)\n",
    "def pipeline(\n",
    "    bq_source: str = \"bq://sara-vertex-demos.beans_demo.large_dataset\",\n",
    "    bucket: str = BUCKET_NAME,\n",
    "    project: str = PROJECT_ID,\n",
    "    gcp_region: str = REGION,\n",
    "    bq_dest: str = \"\",\n",
    "    container_uri: str = \"\",\n",
    "    batch_destination: str = \"\"\n",
    "):\n",
    "    dataset_create_op = gcc_aip.TabularDatasetCreateOp(\n",
    "        display_name=\"tabular-beans-dataset\",\n",
    "        bq_source=bq_source,\n",
    "        project=project,\n",
    "        location=gcp_region\n",
    "    )\n",
    "\n",
    "    training_op = gcc_aip.CustomContainerTrainingJobRunOp(\n",
    "        display_name=\"pipeline-beans-custom-train\",\n",
    "        container_uri=container_uri,\n",
    "        project=project,\n",
    "        location=gcp_region,\n",
    "        dataset=dataset_create_op.outputs[\"dataset\"],\n",
    "        staging_bucket=bucket,\n",
    "        training_fraction_split=0.8,\n",
    "        validation_fraction_split=0.1,\n",
    "        test_fraction_split=0.1,\n",
    "        bigquery_destination=bq_dest,\n",
    "        model_serving_container_image_uri=\"us-docker.pkg.dev/vertex-ai/prediction/sklearn-cpu.0-24:latest\",\n",
    "        model_display_name=\"scikit-beans-model-pipeline\",\n",
    "        machine_type=\"n1-standard-4\",\n",
    "    )\n",
    "    batch_predict_op = gcc_aip.ModelBatchPredictOp(\n",
    "        project=project,\n",
    "        location=gcp_region,\n",
    "        job_display_name=\"beans-batch-predict\",\n",
    "        model=training_op.outputs[\"model\"],\n",
    "        gcs_source_uris=[\"{0}/batch_examples.csv\".format(BUCKET_NAME)],\n",
    "        instances_format=\"csv\",\n",
    "        gcs_destination_output_uri_prefix=batch_destination,\n",
    "        machine_type=\"n1-standard-4\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f052287d-1144-48f9-bec7-81c37320aa90",
   "metadata": {},
   "source": [
    "### Compile and run the pipeline\n",
    "With your pipeline defined, you're ready to compile it. The following will generate a JSON file that you'll use to run the pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "id": "956c4b9c-ee70-4c65-981d-57482a45d920",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jupyter/.local/lib/python3.7/site-packages/kfp/v2/compiler/compiler.py:1266: FutureWarning: APIs imported from the v1 namespace (e.g. kfp.dsl, kfp.components, etc) will not be supported by the v2 compiler since v2.0.0\n",
      "  category=FutureWarning,\n"
     ]
    }
   ],
   "source": [
    "compiler.Compiler().compile(\n",
    "    pipeline_func=pipeline, package_path=\"custom_train_pipeline.json\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cb5aaf0-cdaf-4050-a4eb-a73570f0113a",
   "metadata": {},
   "source": [
    "Next, create a `TIMESTAMP` variable. You'll use this in your job ID:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "id": "ead821be-e97f-4825-8ff7-e0fc37e5dea3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "TIMESTAMP = datetime.now().strftime(\"%Y%m%d%H%M%S\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdae3686-6d9b-40f1-8094-79dff4f260fd",
   "metadata": {},
   "source": [
    "Then define your pipeline job, passing in a few project-specific parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "id": "20e90d2c-aad1-4e65-a080-f6c41c891b4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline_job = aiplatform.PipelineJob(\n",
    "    display_name=\"custom-train-pipeline\",\n",
    "    template_path=\"custom_train_pipeline.json\",\n",
    "    job_id=\"custom-train-pipeline-{0}\".format(TIMESTAMP),\n",
    "    parameter_values={\n",
    "        \"project\": PROJECT_ID,\n",
    "        \"bucket\": BUCKET_NAME,\n",
    "        \"bq_dest\": \"bq://{0}\".format(PROJECT_ID),\n",
    "        \"container_uri\": \"gcr.io/{0}/scikit:v1\".format(PROJECT_ID),\n",
    "        \"batch_destination\": \"{0}/batchpredresults\".format(BUCKET_NAME)\n",
    "    },\n",
    "    enable_caching=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1bb3fdfb-93ba-429c-a14c-5915e3633ef2",
   "metadata": {},
   "source": [
    "Finally, run the job to create a new pipeline execution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "4ff5f20d-f843-4219-af10-79dc97c769db",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:google.cloud.aiplatform.pipeline_jobs:Creating PipelineJob\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:PipelineJob created. Resource name: projects/66861668564/locations/us-central1/pipelineJobs/custom-train-pipeline-20220412152931\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:To use this PipelineJob in another session:\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:pipeline_job = aiplatform.PipelineJob.get('projects/66861668564/locations/us-central1/pipelineJobs/custom-train-pipeline-20220412152931')\n",
      "INFO:google.cloud.aiplatform.pipeline_jobs:View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/custom-train-pipeline-20220412152931?project=66861668564\n"
     ]
    }
   ],
   "source": [
    "# TODO 3: Run the job\n",
    "pipeline_job.submit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b02c0ea2-8e10-4de4-b419-05b28ccaf8a5",
   "metadata": {},
   "source": [
    "After running this cell, you should see logs with a link to view the pipeline run in your console. Navigate to that link. You can also access it by opening your [Pipelines dashboard](https://console.cloud.google.com/vertex-ai/pipelines). This pipeline will take __35-40 minutes__ to run, but you can continue to the next step before it completes. Next you'll learn more about what's happening in each of these pipeline steps."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59f4c2bd-9430-4483-98f4-83bf25b07d91",
   "metadata": {},
   "source": [
    "For further instructions, please refer to the lab manual."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5bce095-3521-4387-bb8a-211453bac906",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-3.m91",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-3:m91"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
